Papers with multimodal agents
SCoPE VLM: Selective Context Processing for Efficient Document Navigation in Vision-Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for document understanding are memory-intensive and impractical for local deployments. |
| Approach: | They propose a document navigation expert that leverages a Chain of Scroll mechanism to selectively and recursively navigate documents, focusing exclusively on relevant segments. |
| Outcome: | The proposed method reduces memory usage and effectively models human-like reading behaviors. |
PresentAgent: Multimodal Agent for Presentation Video Generation (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing methods for generating static slides or text summaries are limited to producing narrated presentations. |
| Approach: | They propose a multimodal agent that transforms long-form documents into narrated presentations. |
| Outcome: | The present agent produces fully synchronized visual and spoken content that closely mimics human-style presentations. |
DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards (2026.findings-eacl)
Copied to clipboard
Aaryaman Kartha, Ahmed Masry, Mohammed Saidul Islam, Thinh Lang, Shadikur Rahman, Ridwan Mahbub, Mizanur Rahman, Mahir Ahmed, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty
| Challenge: | Existing question-answering benchmarks for data visualizations focus on static charts instead of interactive dashboards. |
| Approach: | They propose a benchmark to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |
| Outcome: | The first benchmark explicitly designed to assess how vision-language GUI agents comprehend and interact with real-world dashboards. |
Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perception (2026.findings-acl)
Copied to clipboard
| Challenge: | Small Vision-Language Models (SVLMs) suffer from visual brittleness and poor tool orchestration. |
| Approach: | They propose a supervision-free framework that bootstraps agentic capabilities via Coldstart Reinforcement Learning for SVLMs. |
| Outcome: | The proposed framework improves task accuracy and tool efficiency by 5% and 9%. |
Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation (2025.acl-long)
Copied to clipboard
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark
| Challenge: | Vision-language models struggle to understand text-rich images due to the scarcity of diverse text-only large language data. |
| Approach: | They propose a framework that leverages the coding capabilities of text-only large language models to create synthetic text-rich multimodal data. |
| Outcome: | The proposed framework can generate high-quality instruction-tuning data using Python, HTML, LaTeX and other languages. |
Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are developing but lack external feedback . there is no clear on how to select reward models for agents . |
| Approach: | They propose a benchmark to evaluate agent reward modeling ability in MLLMs . they use multiple dimensions and real-world agent scenarios evaluation . |
| Outcome: | The proposed benchmark evaluates agent performance in multimodal large language models . it covers perception, planning, and safety with 7 scenarios and is highly difficult and high-quality . |
MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct (2025.findings-acl)
Copied to clipboard
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Yongbin Li, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Hamid Alinejad-Rokny, Xiaobo Xia, Jingkuan Song, Fei Huang
| Challenge: | a new framework for image-text instruction data evolution improves MLLM performance . lack of high-quality instruction data remains a major bottleneck in ML modeling . |
| Approach: | They propose a multimodal instruction data evolution framework that iteratively enhances data quality through fine-grained perception, cognitive reasoning, and interaction evolution. |
| Outcome: | The proposed approach improves MLLM performance in nine vision-language tasks while using significantly less data. |
Multimodal Safety Evaluation in Generative Agent Social Simulations (2026.acl-long)
Copied to clipboard
Alhim Adonai Vera Gonzalez, Carlos Hinojosa, Karen Sanchez, Haidar Bin Hamid, Donghoon Kim, Bernard Ghanem
| Challenge: | Recent advances in large language models have enabled generative agents that simulate be-like behavior through natural language interactions. |
| Approach: | They propose a reproducible simulation framework to evaluate generative agents in multimodal scenarios . they use metrics that quantify plan revisions and unsafe-to-safe conversions to evaluate their effectiveness . |
| Outcome: | The proposed framework evaluates generative agents in three aspects: safety improvement over time, detection of unsafe activities across social contexts, social dynamics and acceptance rates. |